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Abstract 

For a large number of random constraint satisfaction problems, such as random k-SAT and random graph and 
hypergraph coloring, there are very good estimates of the largest constraint density for which solutions exist. Yet, 
all known polynomial-time algorithms for these problems fail to find solutions even at much lower densities. To 
understand the origin of this gap we study how the structure of the space of solutions evolves in such problems as 
constraints are added. In particular, we prove that much before solutions disappear, they organize into an exponential 
number of clusters, each of which is relatively small and far apart from all other clusters. Moreover, inside each cluster 
most variables are frozen, i.e., take only one value. The existence of such frozen variables gives a satisfying intuitive 
explanation for the failure of the polynomial-time algorithms analyzed so far. At the same time, our results establish 
rigorously one of the two main hypotheses underlying Survey Propagation, a heuristic introduced by physicists in 
recent years that appears to perform extraordinarily well on random constraint satisfaction problems. 

1 Introduction 

For a number of random Constraint Satisfaction Problems (CSP), by now very good estimates are available for the 
largest constraint density (ratio of constraints to variables) for which typical problems have solutions. For example O, 
a random graph of average degree d is with high probabilityQ fc-colorable if d < {2k — 2) ln(/s — 1), but w.h.p. non- 
fc-colorable if d > {2k — 1) In k. This implies that for every d > 0, w.h.p. the chromatic number of a random graph 
with average degree d is either kd or kd + 1, where kd is the smallest integer k such that d < 2k\n k. 

Algorithmically, it is very easy to get a factor-2 approximation for the graph coloring problem on random graphs. 
The algorithm '"repeatedly pick a random vertex and assign it a random available color" will w.h.p. succeed in coloring 
a random graph of average degree d if originally each vertex has 2kd available colors. Alternatively, k colors suffice 
when d < k\nk. In spite of significant efforts over the last 30 years, no improvement has been made over this 
trivial algorithm. Specifically, no polynomial-time algorithm is known that fc-colors random graphs of average degree 
d = (1 + e)fc In fc, for some fixed e > and arbitrarily large k. 

In the random fc-SAT problem one asks if a random fc-CNF formula, Fk{n, m), with n variables and m clauses 
is satisfiable. It is widely believed that the probability that such a formula is satisfiable exhibits a sharp threshold. 
Specifically, the Satisfiability Threshold Conjecture asserts that = r^, for all fc > 3, where 

Tk = sup{r : Fk{n, rn) is satisfiable w.h.p.} , 
r*f. = inf {r : rn) is unsatisfiable w.h.p.} . 

It is easy to see that < 2^ In 2, since the probability that at least one assignment satisfies Fh{n, rn) is bounded 
by 2"(1 — 2^'"') ' ", a quantity that tends to for r > 2^ In 2. Recently, it was shown that random fc-CNF formulas have 
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satisfying assignments for densities very close to this upper bound lH]. Specifically, it was proven that for all fc > 3, 

,.>.M„2-<^L±i^. (1) 

As for the fc-coloring problem, the lower bound on the largest density for which solutions exist w.h.p. is non- 
constructive, based on the second moment method. Here, the gap relative to algorithms is ever greater: no polynomial 
algorithm is known that finds satisfying assignments in a random fc-CNF formula when r = Lo{k) 2^ /k, for any func- 
tion ijj{k) — *■ oo (arbitrarily slowly). In Table [H we illustrate this gap for some small values of k. For fc = 3, the 
upper bound on comes from 1121 . while for A: > 3 from ifTTHTSl . The best algorithmic lower bound for fc = 3 is 
from H?!, while for fc > 3 it is from lfT4l . 

Similar results (and gaps) exist for a number of other constraint satisfaction problems, such as random NAE fc-S AT 
and hypergraph 2-coloring, regular random graph coloring, random Max fc-SAT, and others (for example, see [4|). 
Indeed, this phenomenon seems to occur in nearly all random CSP in which the underlying constraint graph is sparse 
and random, making it natural to ask if there is a common underlying cause. (The bipartite graphs where constraints 
are adjacent to the variables they bind are also known as factor graphs.) 

As it turns out, sparse random CSP have been systematically studied by physicists in the past few decades under the 
name "mean-field diluted spin-glasses". Spins here are the variables (reflecting the notion that variables have small, 
discrete domains), the term glass refers to the fact that the system has not been allowed to relax to a configuration in 
which spins interact in a mutually agreeable way (reflecting that different constraints prefer different values for the 
variables), diluted refers to the fact that the factor graph is sparse (reflecting that each spin interacts with only a few 
other spins), while "mean field" refers to the fact that the factor graph is random, i.e., there is no underlying geometry 
mandating the interactions. The interest in such "unphysical" systems is partly motivated by the fact that in many 
statistical mechanics problems where the variables do lie on a lattice such as U^, the effect of the underlying geometry 
vanishes for d sufficiently large (but finite). 

Perhaps more surprising is the fact that in the last few years, motivated by ideas developed for the study of ma- 
terials, physicists have put forward a hypothesis for the origin of the aforementioned algorithmic gap in random CSP 
and, most remarkably, a method for overcoming it. Specifically, Mezard, Parisi, and Zecchina 1221 developed an ex- 
tremely efficient algorithm, called Survey Propagation (SP), for finding satisfying assignments of random formulas in 
the satisfiable regime. For example, their algorithm typically finds a satisfying truth assignment of a random 3-CNF 
formula with n = 10^ variables and A.25n clauses in minutes (and appears to scale as 0{n log n)). No other algorithm 
practically solves formulas of such density with n = lO"'. 

Our original motivation for this work was to see if some of the physically-motivated ideas underlying SP can be 
proven mathematically. More generally, we believe that understanding the geometry of the space of satisfying truth 
assignments in random formulas is essential for understanding the behavior of algorithms on them. This is particularly 
true for the case of random-walk type algorithms, which we view as the first natural class to target armed with such an 
understanding and for which very little is known rigorously, with the notable exception of Q . 

We make significant progress towards this goal by proving that already much below the satisfiability threshold, 
the set of satisfying assignments fragments into exponentially many connected components. These components are 
relatively small in size, far apart from one another, and inside each one the majority of variables are "frozen", i.e., 
take only one value. As the formula density is increased towards the threshold, the fraction of frozen variables in 
each component increases, causing the connected components to decrease in volume and grow further apart from one 
another 

Our results are in perfect agreement with the picture put forward by the physicists. Moreover, as we discuss 
below, the existence of frozen variables provides a good explanation for the origin of the barrier faced by all analyzed 
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algorithms on random CSP, i.e., "local", DPLL-type algorithms. Finally, we show that one of the two main assumptions 
underlying SP regarding the structure of the set of solutions is essentially correct. This brings us closer to a rigorous 
analysis of SP and answers affirmatively the main open question raised by Maneva, Mossel and Wainwright in |fT9l . 

Specifically, we prove that for all fc > 9, the connected components of the set of satisfying assignments of random 
formulas have non-trivial cores, as assumed by SP (see Definition [T2]|. We point out that it is not clear whether this 
is true for small k. Indeed, |fT9l gave experimental evidence that for fc = 3, random formulas do not have non-trivial 
cores. As we will see, our methods also give some evidence in that direction. This gives additional motivation for the 
"core-like" objects introduced in |fT9l whose existence would relate to the success of SP for small k (we discuss this 
point further in Section |4TT] i. 

In the next section we give an informal discussion relating the performance of DPLL-type algorithms on random 
formulas to notions such as Gibbs sampling and long-range correlations. This is meant to provide intuition for the 
empirical success of SP and motivate our results. We emphasize that while both the discussion and the results are 
about random fc-SAT, this is not strictly necessary: our ideas and proofs are quite generic, and should generalize 
readily to many other random CSP, e.g., graph coloring. 

1.1 DPLL algorithms, Belief Propagation, and Frozen Variables 

Given a satisfiable formula F on variables vi,V2, ■ ■ ■ ,Vn it is easy to see that the following simple procedure samples 
uniformly from the set of all satisfying assignments of F: 

Start with the input formula F 

For i = 1 to n do: 

1 . Compute the fraction, pi, of satisfying assignments of the current formula in which Vi takes the value 1 . 

2. Set Vi to 1 with probability Pi and to otherwise. 

3. Simplify the formula. 

Clearly, the first step in the loop above is meant only as a thought experiment. Nevertheless, it is worth making 
the following two observations. The first is that if we are only interested in finding some satisfying assignment, as 
opposed to sampling a uniformly random one, then we do not need to compute exact marginals. For example, if we 
use the rule of always setting Vi to 1 iff pi > 1/2, then it is enough to ensure that if a variable takes the same value 
X in all satisfying assignments, x is the majority value in its computed marginal. The second observation is that the 
order in which we set the variables does not need to be determined a priori. That is, we can imagine that in each step 
we compute marginals for all remaining variables and that for each marginal we have an associated confidence. To 
improve our chances of avoiding a fatal error, we can then set only the variable for which we have highest confidence. 

The above two elementary observations in fact capture all algorithms that have been analyzed so far on random 
formulas (and, in fact, most DPLL-type algorithms used in practice). Observe, for example, that both the unit-clause 
and the pure literal heuristics follow immediately from the above considerations. In the case of unit-clause, the 
participation of a variable w in a unit clause c allows us to infer its marginal with perfect confidence and thus setting 
V is an "obvious" choice. In the case of a pure literal £, again we can infer with certainty the majority marginal of the 
underlying variable v (it is the value that satisfies £). In the absence of such obvious choices, all DPLL-type algorithms 
attempt to identify a variable whose marginal can be determined with some confidence. For example, below are the 
choices made in the absence of unit clauses and pure literals by some of the algorithms that have been analyzed on 
random 3-CNF formulas. In order of increasing performance: 

UNIT-CLAUSE ifTOl : select a random variable and assign it a random value. 

3-CLAUSE MAJORITY ||9l: select a random variable and assign it its majority value among the 3-clauses. 
SHORT-CLAUSE llT4l : select a random shortest clause c, a random variable v in c, and set v so as to satisfy c. 
HAPPIEST LITERAL lfT6l : Satisfy a literal that appears in most clauses. 
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Each of the above heuristics attempts to compute marginals based on a different set of evidence, the content of 
which ranges from completely empty ifTOl . to considering all the clauses containing each variable ifTSll . Correspond- 
ingly, the largest density for which these algorithms succeed on random 3-CNF formulas ranges from 8/3 for IfTOl 
to 3.42 for |fT6l . UNIT-CLAUSE, in fact, succeeds for every k as long as r < 2''/k and, as we mentioned earlier, no 
algorithm is known to beat this bound asymptotically. Given that improving upon the empty set of evidence is rather 
easy, it is tempting to think that by considering a larger set of evidence for each variable one can do significantly 
better. For example, consider an algorithm Ad which computes a marginal for each variable v based on the clauses 
that appear in the depth-d neighborhood of v in the factor graph. One could hope that as d grows, such an algorithm 
would do well, perhaps even reach the satisfiability threshold. 

Physicists say it is not so. The hope that local algorithms could do well on random formulas rests on the pre- 
sumption that the influence exerted on a variable v by other variables, diminishes rapidly with their distance from v in 
the factor graph. That is, that there are no "long range correlations" in random formulas, so that the joint probability 
distribution of a random finite subset of the variables should be, essentially, the product of their marginals. 

Unfortunately, the existence of connected components of satisfying assignments (clusters) with numerous frozen 
variables can induce long-range correlations among the variables, eliminating such hopes. For example, if one consid- 
ers any fixed set of variables, at sufficiently large densities their joint behavior over the set of satisfying assignments 
is dominated by a small number of connected components: those in which most of them are frozen, freeing up other 
variables to take multiple values and amplify the contribution of that particular joint collection of values to the vari- 
able marginals. In other words, assuming that the variables in the boundary of a variable's neighborhood behave 
independently with respect to the rest of the formula, can be very far off from the truth. 

To overcome the above issue, physicists hypothesized that the above clustering is the only significant source of 
long-range correlations. (Very) roughly speaking, this amounts to modeling each connected component of satisfying 
assignments as a subcube that results by selecting a large fraction of the variables and freezing them independently at 
random, while leaving the rest (largely) free. Our results imply that this simplified view of clusters is not very far off 
the truth. 

1.2 Organization 

In the next section we give mathematical statements of our main results, regarding the existence of exponentially many 
well-separated clusters and the existence of frozen variables in every cluster. In Section [3] we outline the proof of the 
results on the existence of clusters and explain their relationship to the work of Mora, Mezard, and Zecchina ||20ll2Tl . 
In Section |4] we provide some background on Survey Propagation and explain how our results on frozen variables 
relate to the implicit hypothesis made by Mezard, Parisi, and Zecchina in their derivation of Survey Propagation 1221 . 
and how our results answer the main open question of Maneva, Mossel and Wainwright ||T9l . In Section|5]we introduce 
the probabilistic setup for our analysis and in Section|6]we discuss how it relates to the case fc < 8 and to the "planted 
assignment" model. Proving our main result on the existence of frozen variables boils down to a question in large 
deviations developed in Section|7]and an associated multi-dimensional optimization problem, resolved in Section|8] 

2 Statement of Results 

We first need to introduce some definitions. Throughout, we assume that we are dealing with a CNF formula F, 
defined over variables X ^ xi, . . . , x„, and we let S{F) C {0, 1}" denote the satisfying assignments of F. 

Definition 1. The diameter of an arbitrary set X C {0, 1}" is the largest Hamming distance between any two 
elements of X. The distance between two arbitrary sets X^Y (- {0, 1}", is the minimum Hamming distance between 
any x & X and any y ^ Y. The clusters of a formula F are the connected components of S{F) when x,y {0, 1}" 
are considered adjacent if they have Hamming distance 1. A cluster-region is a non-empty set of clusters. 

Our first set of results is captured by Theorems|2]and|3]below, which build upon the work in ||5]|20]|2T|. We discuss 
the relationship between our results and those in Il20ll21l in Section |33] 

Tlieorem 2. For every k > 8, there exists a value of r < and constants < Pk < 1/2 and > such that 
w.h.p. the set of satisfying assignments of Fk{n, rn) consists of 2'^''" non-empty cluster-regions, such that 
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1. The diameter of each cluster-region is at most akn. 



2. The distance between every pair of cluster- regions is at least Pk^. 

In other words, for all /c > 8, at some point below the satisfiability threshold, the set of satisfying assignments 
consists of exponentially many, well-separated cluster-regions. The picture suggested by Theorem|2]comes in sharper 
focus for large k. In particular, for sufficiently large fc, sufficiently close to the threshold, the cluster regions be- 
come arbitrarily small and maximally far apart (while remaining exponentially many). The following result gives a 
quantitative version of this fact. 

Theorem 3. For any < (5 < 1/3, i/r = (1 - 6)2'' In 2, then for all k > kQ{5), Theorem]2\holds with 

It is worth noting that, as we will show shortly. 

Remark 4. Theorems\2\and\3\remain valid for any definition of clusters in which a pair of assignments are deemed 
adjacent whenever their distance is at most f{n) for some function f{n) = o{n). 

Our main result in this paper comes from "looking inside" clusters and proving the existence of variables which 
take the same value in all the truth assignments of a cluster More formally, 

Deflnition 5. The projection of a variable Xi over a set of satisfying assignments C, denoted as iiiiC), is the union of 
the values taken by Xi over the assignments in C. IfiTi{C) ^ {0, 1} we say that Xi is frozen in C. 

No previous results were known about the existence of frozen variables. The existence of such variables is a 
fundamental underpinning of the approximations implicit in the Survey Propagation algorithm. To prove that random 
formulas have frozen variables we, in fact, prove that random formulas have non-trivial cores for all fc > 9, thus also 
answering the main question of |fT9l (we postpone the definition of cores until Section|4]i. A strength of our approach 
is that it allows us to prove not just the existence, but the pervasiveness of frozen variables. Specifically, Theorem|6] 
below asserts that for sufficiently large fc, as we approach the satisfiabihty threshold, the fraction of frozen variables 
in every single cluster gets arbitrarily close to 1 . 

Theorem 6 (Main Resuh). For every a > and all k > k^la), there exists < r^, such that for all r > c^, w.h.p. 
every cluster of Fk{n, rn) has at least (1 — a)n frozen variables. As fc grows. 



By taking a = 1/2 in Theorem|6]we see that for sufficiently large fc, every cluster already has a majority of frozen 
variables at r = (4/5 + (5fe)2'^ In 2, with 6k 0, i.e., for a constant fraction of the satisfiable regime. More generally, 
Theorem|6]asserts that as fc grows and the density approaches the threshold, clusters shrink in volume and grow further 
apart by having smaller and smaller internal entropy (more frozen variables). 

The analysis that establishes Theorem|6]also allows us to show 

Corollary 7. For every fc > 9, there exists r < such that w.h.p. every cluster of Fk{n, rn) has frozen variables. 

It remains open whether frozen variables exist for fc < 8. As we mentioned above, |fT9l reported experimental 
evidence suggesting that frozen variables do not exist for fc = 3. We will see that our proof also gives evidence in this 
direction for small values of fc. 

3 Clustering: Proof Sketch and Related Work 

There are two main ingredients for proving Theorems |2] and [3] The first one excludes the possibility of pairs of truth 
assignments at certain Hamming distances. 
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3.1 Forbidden distances and their implications for clustering 

It is easy to show, see e.g., ||2l, that the expected number of pairs of satisfying assignments in Fk {n, rn) with Hamming 
distance z is at most K{z/n, k, r)", where 



A(q;, k, r) 



2(1- 21-'= + 2-'=(l 



Therefore, if for some k, r and z = an we have A{a, k, r) < 1, it immediately follows by the union bound that 
w.h.p. in Fk{n, rn) no pair of satisfying assignments has distance z. This observation was first made and used in 1201 . 
In Figure [T] we draw the function A (upper curve), and a related function At (lower curve, to be discussed shortly), 
for a £ [0,3/4] with fc = 8 and r = 169. Recall that, by the results in lO, Fs{n,lQ9n) is w.h.p. satisfiable 
and, thus, excluding the possibility of satisfying pairs at certain distances is a non-vacuous statement. We see that 
A{a, 8, 169) < 1 for a e [0.06, 0.26] U [0.68, 1], implying that w.h.p. in Fg{n, 169n) there is no pair of satisfying 
assignments with Hamming distance an for such values of a. 




Figure 1: Upper curve A{a, 8, 169) and lower curve A;,(a, 8, 169) for a E [0, 3/4]. 

Establishing that there exists a distance z such that there are no pairs of assignments at distance z immediately 
implies an upper bound on the diameter of every cluster. This is because if a cluster C has diameter d, then it must 
contain pairs of solutions at every distance 1 < t < d. To see this, take any pair cri , (72 G C that have distance d, 
any path from cti to (T2 in C, and observe that the sequence of distances from cri along the vertices of the path must 
contain every integer in {1, . . . , d}. Therefore, if Makis = Ak,r = infja : A{a, k, r) < 1}, then w.h.p. every cluster 
in Fk {n, rn) has diameter at most An. 

If we can further prove that A(a, fc, r) < 1 in an interval (a, /?), then we can immediately partition the set of 
satisfying assignments into well-separated regions, as follows. Start with any satisfying assignment a, let C be its 
cluster, and consider the set R{C) C {0, 1}" of truth assignments that have distance at most an from C and the set 
B{C) C {0, 1}" of truth assignments that have distance at most /3?7 from R{C). Observe now that the set B{C)\R{C) 
cannot contain any satisfying truth assignments, as any such assignment would be at distance an < d < (3n from some 
assignment in C. Thus, the set of satisfying assignments in R{C) is a union of clusters (cluster-region), all of which 
have distance at least /3n from any cluster not in the region. Repeating this process until all satisfying assignments 
have been assigned to a cluster region gives us exactly the subsets of Theorems |2] and |3] Moreover, note that this 
arguments bounds the diameter of each entire cluster-region, not only of each cluster, by an. 

Remark 8. The arguments above remains valid even if assignments are deemed adjacent whenever their distance is 
bounded by f{n), for any f{n) = o{n). As a result, Theorems^and\3\remain valid as stated for any definition of 
clusters in which assignments are deemed to belong in the same cluster if their distance is o{n). 
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3.2 Establishing exponentially many clusters 

Proving the existence of exponentially many non-empty cluster regions requires greater sophistication and leverages 
in a strong way the results of 151. This is because having A(q;, fc, r) > 1 for some a, k, r does not imply that pairs of 
satisfying assignments exist for such a,k,r: in principle, the behavior of A could be determined by a tiny minority 
of solution-rich formulas. Hence the need for the second moment method ||2]|5|. Specifically, say that a satisfying 
assignment is balanced if its number of satisfied literal occurrences is in the range km/ 2 ± ^/n, and let X be the 
number of balanced assignments in Fk{n, rn). In [51, it was shown that E[X]^ = Ah(l/2, fc, r)" and 

EfAT^l < C X max At (a, fc,r)" , 

"6[0,1] 

for some explicit function Ah and constant C = C(fc) > 0. It was also shown that for all r < 2*^ In 2 — fc, the maximum 
of Ah occurs at a = 1/2, implying that for such fc, r we have E[Ar^] < C x E[X]^. By the Payley-Zigmund inequality, 
this last fact implies that for any t < K[X], 

In 15], inequality (|2]i was applied with t = 0, per the "second moment method", estabhshing that for r <2^ In 2— fc, 
Jfc(n, rn) has at least one (balanced) satisfying assignment with probability at least 1/C. Taking t = E[X]/poly(n), 
implies that X is within a polynomial factor of its expectation Ah(l/2, fc, r)"/^, also with constant probability. Since 
the property "has more than q satisfying assignments" has a sharp threshold iflBll , this assertion implies that for every 
r < 2*^ hi 2 — fc, F}^{n, rn) has at least Ah(l/2, fc, r)"/^/poly(7i) satisfying assignments w.h.p. 

To prove that there are exponentially many clusters, we divide the above lower bound for the total number of 
satisfying assignments with the following upper bound for the number of truth assignments in each cluster-region. 
Recall that A = Afe,,, = inf{Q! : A(a, fc, r) < 1} and let 

g{k,r) = max A(q;, fc,r) . 

a6[0,A] 

If B is the expected number of pairs of truth assignments with distance at most An in Fk{n, rn), it follows that 
B < poly(?i) X g(k, r)", since the expected number of pairs at each distance is at most A(a, fc, r)" and there are 
no more than n + 1 possible distances. By Markov's inequality, this implies that w.h.p. the number of pairs of truth 
assignments in Fk{n, rn) that have distance at most An is poly(n) x g{k, r)". Recall now that w.h.p. every cluster- 
region in Fk (n, rn) has diameter at most An. Therefore, w.h.p. the total number of pairs of truth assignments in each 
cluster-region is at most poly(n) x g(k, r)". Thus, if g(fc, r) < A(,(l/2, fc, r), we can conclude that Fk{n, rn) has at 
least 

^ , , , , ^ Ah(l/2,fc,r ^^"/' 
l/poly(n) X 

cluster-regions. Indeed, the higher of the two horizontal lines in Figure [T]highlights that 17(8, 169) < Ah(l/2, 8, 169). 
From the discussions in this section we see that to establish Theorem|2]it suffices to prove the following. 

Theorem 9. For every fc > 8, there exists a value of r < r^ and constants < /3fc < 1/2 and > such that 
A(a, fc, r) < I for all a G (a^, /3fc) and 



log2 



Ah(l/2,fc,r)V/' 



g{k,r) 

In particular, for any < 5 < 1/3 and all fc > ko{5), ifr = {\ — 5)2^ In 2, we can take 



Specifically, in Section|9]we will prove the claims in Theorem|9]regarding a^, /3k, while in Section [TOl we prove 
the claims regarding Cfc. 
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3.3 Related Work 



The observation that if A(a, k,r) < 1 then w.h.p. Fk {n, rn) has no pairs of satisfying assignments at distance an was 
first made and used in ||20l . Moreover, in IJTl the authors gave an expression Ai{a, k, r) for the expected number of 
locally maximal pairs of satisfying assignments at each distance, where a pair a, r is locally maximal if there is no 
variable which has value in cr and flipping its value in both a and r yields a new pair of satisfying assignments. (If 
a formula has a pair of satisfying assignments at distance d, then it always has a locally maximal pair at distance d). 
Clearly, Ag{a, k, r) < A(a, k, r) always, but for large k and r — 0(2'"') the difference is minuscule for all a. 

The connection between A(a, k,r) < 1 in an interval and "clustering" was also first made in ||20| . Unfortunately, 
in 1201 no concrete definition of clusters was given and, certainly, no scheme for grouping clusters into well-separated 
cluster regions (clusters need not be well separated themselves). Besides these simple clarifications, our minor con- 
tribution regarding clustering lies in giving rigorous bounds on the diameter and distance of the cluster regions. The 
novel one, as we discuss below, lies in establishing the existence of exponentially many cluster regions. 

Additionally, in [20 2 IJ the authors derive an expression for the second moment of the number of pairs of balanced 
assignments at distance an, for each a E [0, 1]. Whenever, for some a, fc, r, the dominant contribution to this second 
moment comes from uncorrected pairs of pairs of balanced assignments, this implies that with constant probability 
Fk{n, rn) contains at least one (balanced) pair of assignments at distance an. We note that determining the dominant 
contribution to the above second moment rigorously, given a,k,r, is a highly non-trivial problem which the authors 
tackle numerically for small k, and heuristically for general k, i.e., they make a guess for the locus of the maximizer 
In particular, this "fourth moment" optimization problem is much harder than the already complicated second moment 
analysis of [5|. 

Finally, the authors prove that the property "has a pair of satisfying assignments at distance has a sharp threshold, 
thus boosting their constant probability result for having a pair of satisfying assignments at a given distance to a high 
probability one. To the best of our understanding, these three are the only results established in 11211 . Combined, they 
imply that for every fc > 8, there is r < and constants ak < (3k < Ck < 1/2 < dk, such that in Fk{n, rn): 

• W.h.p. every pair of satisfying assignments has distance either less than akn or more than Pkn. 

• For every d E [ck,dk] ■ n, w.h.p. there is a pair of truth assignments that have distance d. 

We note that even if the maximizer in the second moment computation was determined rigorously and coincided with 
the heuristic guess of 1211, the strongest statement that can be inferred from the above two assertions in terms of 
"establishing clustering" is: for every fc > 8, there is r < r^, such that w.h.p. S{Fk{n, rn)) has at least two clusters. 

In contrast, our Theorem |2] establishes that w.h.p. S{Fk{n,rn)) consists of exponentially many, well-separated 
cluster regions, each region containing at least one cluster. Additionally, Theorem|3]establishes that as fc grows and r 
approaches the threshold, these regions grow maximally far apart and their diameter vanishes. 

4 Frozen Variables: Survey Propagation and Related Work 

For a cluster C, the string tt{C) = tti{C), tt2{C), . . . , 7r„(C) is the projection of C and we will use the convention 
{0, 1} = *, so that 7r(C) E {0, 1, *}". Imagine for a moment that given a formula F we could compute the marginal 
of each variable over the cluster projections, i.e., that for each variable we could compute the fraction of clusters in 
which its projection is 0, 1, and *. Then, as long as we never assigned 1 — a; to a variable which in every cluster was 
frozen to the value x, we are guaranteed to find a satisfying assignment: after each step there is at least one cluster 
consistent with our choices so far 

Being able to perform the above marginalization seems quite far fetched given that even if we are handed a truth 
assignment cr in a cluster C, it is not at all clear how to compute tt{C) in time less than |C|. Survey Propagation (SP) 
is an attempt to compute marginals over cluster projections by making a number of approximations. One fundamental 
assumption underlying SP is that, unlike the marginals over truth assignments, the marginals over cluster projections 
essentially factorize, i.e., if two variables are far apart in the formula, then their joint distribution over cluster pro- 
jections is essentially the product of their cluster projection marginals. Determining the validity of this assumption 
remains an outstanding open problem. 
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The other fundamental assumption underlying SP is that approximate cluster projections can be encoded as the 
solutions of a CSP whose factor graph can be syntactically derived from the input formula. Our results are closely 
related to this second assumption and establish that, indeed, the approximate cluster projections used in SP retain a 
significant amount of information from the cluster projections. To make this last notion concrete and enhance intuition, 
we give below a self-contained, brisk discussion of Survey Propagation. For the sake of presentation this discussion is 
historically inaccurate. We attempt to restore history in Section|4T| 

As we said above, even if we are given a satisfying assignment a, it is not obvious how to determine the projection 
of its cluster C((t). To get around this problem SP sacrifices information in the following manner. 

Definition 10. Given a string x G {0, 1, *}", a variable .t,; is free in x if in every clause c containing Xi orxi, at least 
one of the other literals in c is assigned true or *. 
We will refer to the following as a 

coarsening-step: if a variable is free, assign it *. 

Given x,y Cz {0, 1, say that x is dominated by y, written x ^ y, if for every i, either Xi = yi or yi — *. 

Consider now the following process: 

start at a and apply coarsening until a fixed point is reached. 

Lemma 11. For every formula F and truth assignment a G S{F), there is a unique coarsening fixed point w{a). If 
(Ti, (72 belong to the same cluster C, then w{<Ti) = w{a2) ^ "^{C)- 

Proof. Trivially, applying a coarsening step to a string x produces a string y such that x <y. Moreover, if x,; was free 
in X, then yi will be free in y. As a result, if both y,z E {0, 1, are reachable from x E {0, 1, *}" by coarsening 
steps, so is the string that results by starting at x, concatenating the two sequences of operations and removing all but 
the first occurrence of each coarsening step. This implies that there is a unique fixed point w;(a;) for each a; e {0, 1 , 
under coarsening. Observe now that if a, a' E S{F) differ only in the z-th coordinate, then the i-th variable is free in 
both cr, <t' and coarsening it in both yields the same string r. By our earlier argument, w{(j) = ■w{t) = w{(j') ~ wc, 
where C C S{F) is the cluster containing a, a' . Considering all adjacent pairs in C, we see that wc ^ '''(C)- CH 

Definition 12. The core of a cluster C is the unique coarsening fixed point of the truth assignments in C. 

By Lemma[TT] if a variable takes either the value or the value 1 in the core of a cluster C, then it is frozen to that 
value in C. To prove Theorem|6]we prove that the core of every cluster has many non-* variables. 

Tfieorem 13. For any a > 0, let kQ{a) and cf. be as in Theorem^ Ifk > andr > cJJ, then w.h.p. the coarsening 
fixed point of every a E S{Fk (n, rn)) contains fewer than a ■ n variables that take the value *. 

To prove Theorem [13] (which implies Theorem |6l) we derive sharp bounds for the large deviations rate function 
of the coarsening process applied to a fixed satisfying assignment. As a result, we also prove that in the planted- 
assignment model the cluster containing the planted assignment already contains frozen variables at r ~ (2*^/^) In k. 
Also, we will see that our proof gives a strong hint that for small values of k, such as fc = 3, for all densities in the 
corresponding satisfiable regime, most satisfying assignments do converge to • • • , *) upon coarsening. 

We can think of coarsening as an attempt to estimate the projection of C{cr) by starting at a and being somewhat 
reckless. To see this, consider a parallel version of coarsening in which given x E {0, 1, *}" we coarsen all free 
variables in it simultaneously. Clearly, the first round of such a process will only assign * to variables whose projection 
in C{a) is indeed Subsequent rounds, though, might not: a variable v is deemed free, if in every clause containing 
it there is some other variable satisfying the clause, or a variable assigned *. This second possibility is equivalent 
to assuming that the *-variables in the clauses containing v, call them F^, can take joint values that allow v to not 
contribute in the satisfaction of any clause. In general formulas this is, of course, not a valid assumption. On the other 
hand, the belief that in random formulas there are no long-range correlations among the non frozen variables of each 
cluster makes this is a reasonable statistical assumption: since the formula is random, the variables in Ty are probably 
far apart from one another in the factor graph that results after removing the clauses containing v. Thus, indeed, any 
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subset of variables of that do not co-occur in a clause should be able to take any set of joint values. Our results 
can be seen as evidence of the utility of this line of reasoning, since we prove that for sufficiently large densities, 
the coarsening fixed point of a satisfying assignment is never (*,..., *). Indeed, as we approach the satisfiabihty 
threshold, the fraction of frozen variables in it tends to 1 . 

Of course, while the core of a cluster C can be easily derived given some a G C, such a cr is still hard to come by. 
The last leap of approximation underlying SP is to define a set Z{F) C {0,1, *}" that includes all cluster cores, yet is 
such that membership in Z{F) is "locally checkable", akin to membership in S{F). Specifically, 

Definition 14. A string x G {0, 1, *}" is a cover of a CNF formula F if: (i) under x, every clause in F contains a 
satisfied literal or at least two *, and (ii) every free variable in x is assigned i.e., x is ^-maximal. 

Cores trivially satisfy (ii) as fixed points of coarsening; it is also easy to see, by induction, that any string that results 
by applying coarsening steps to a satisfying assignment satisfies (i). Thus, a core is always a cover. At the same time, 
checking whether x £ {0,1,*}" satisfies (i) can be done trivially by examining each clause in isolation. For (ii) it is 
enough to check that for each variable v assigned or 1 in x, there is at least one clause satisfied by v and dissatisfied 
by all other variables in it. Again, this amounts to n simple checks, each check done in isolation by considering the 
clauses containing the corresponding variable. The price we pay for dealing with locally-checkable objects is that the 
set of all covers Z{F) can be potentially much bigger than the set of all cores. For example, (*,• •• , *) is always a 
cover, even if F is unsatisfiable. 

The Survey Propagation algorithm can now be stated as follows. 

Repeat until all variables are set: 

1. Compute the marginals of variables over covers. 

2. Select a variable with least mass on * and assign it the 0/1 value on which it puts most mass. 

3. Simplify the formula. 

The computation of marginals over covers in the original derivation 1221123)1 of SP was, in fact, done via a message 
passing procedure that runs on the factor graph of the original formula rather than a factor graph encoding covers 
(more on this in Section|4]TJ. Also, in Il22l|23l , if a configuration is reached in which all variables put (nearly) all their 
mass on *, the loop is stopped and a local search algorithm is invoked. The idea is that when such a configuration is 
reached, the algorithm has "arrived" at a cluster and finding a solution inside that cluster is easy since only non-frozen 
variables remain unset. 

4.1 Related Work 

The original presentation of Survey Propagation motivated the algorithm in terms of a number of physical notions 
(cavities, magnetic fields, etc.). Specifically, the algorithm was derived by applying the "cavity method" within a 
"1-step Replica Symmetry Breaking" scheme, with no reference whatsoever to notions such as cluster projections, 
cores, or covers (in fact, even clusters where only specified as the connected components that result when satisfying 
assignments at "finite Hamming distance" are considered adjacent). On the other hand, a very definitive message- 
passing procedure was specified on the factor graph of the original formula and the computer code accompanying 
the paper and implementing that procedure worked spectacularly well. Moreover, a notion foreshadowing cores was 
included in the authors' discussion of "Warning Propagation". 

Casting SP as an attempt to compute marginals over cores was done independently by Braunstein and Zecchina 
in m and Maneva, Mossel, and Wainwright in |fT9l . In particular, in both papers it is shown that the messages 
exchanged by SP over the factor graph of the input formula are the messages implied by the Belief Propagation 
formalism 16] applied to a factor graph encoding the set of all covers. The first author and Thorpe H) have additionally 
shown that for every formula F, there is a factor graph Gf encoding the set of F's covers which inherits the cycle 
structure of F's factor graph, so that if the latter is locally tree-like so is Gf- 

In fT9l , the authors give a number of formal correspondences between SP, Markov random fields and Gibbs 
sampling and note that a cover a G {0, 1, can also be thought of as partial truth assignment in which every 
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unsatisfied clause has length at least 2, and in which every variable v assigned or 1 has some clause c for which it is 
essential in a, i.e., v satisfies c but all other variables in c are set opposite to their sign in c. This last view motivates a 
generalization of SP in which marginals are computed not only over covers, but over all partial assignments in which 
every unsatisfied clause has length at least 2, weighted exponentially in the number of non-essential 0/1 variables and 
the number of ^-variables. One particular motivation for this generalization is that while SP appears to work very well 
on random 3-CNF formulas, [ |T9l gives experimental evidence that such formulas do not have non-trivial cores, i.e., 
upon coarsening truth assignments end up as (*,..., This apparent contradiction is reconciled by attributing the 
success of SP to the existence of "near-core" strings allowed under the proposed generalization. 

While |[T9| provided a framework for studying SP by connecting it to concrete mathematical objects such as cores 
and Markov random fields, it did not provide results on the actual structure of the solution space of random fc-CNF 
formulas. Indeed, motivated by the experimental absence of cores for fc = 3, the authors asked whether random 
formulas have non-trivial cores for any fe. Our results, establish a positive answer to this question for all fc > 9. 



5 The Probabilistic Framework 

Theorem|6]follows from Theorem [T3] and Lemma[TT] To prove Theorem[T3]we say that a satisfying assignment a is 
a-coreless if its coarsening fixed point w{a) has at least an ^-variables. Let X be the random variable equal to the 
number of a-coreless satisfying assignments in a random fe-CNF formula Fk{n, rn). By symmetry, 

E[X] = Pr[(T is a-coreless | a is satisfying] • Pr[a is satisfying] (4) 

o-e{o,i,}" 

= 2" • ( 1 - ^ j • Pr[0 is a-coreless | is satisfying] . (5) 

Observe that conditioning on "0 is satisfying" is exactly the same as "planting" the solution, and amounts to 
selecting the m = rn random clauses in our formula, uniformly and independently from amongst all clauses having 
at least one negative literal. We will see that for every fe > 3, there exists such that 

{1 — o(l) if r < 
' (6) 
o(l) if r > tJJ . 

In particular, we will see that tj. ^ (Z'^/fc) hi fe. We find it interesting (and speculate that it's not an accident) that all 
algorithms that have been analyzed so far work for densities below t]^. More precisely, all analyzed algorithms set each 
variable v by considering only a subset of the not yet satisfied clauses containing v and succeed for some r < c2^ /k, 
where c depends on the choice of subset. 

To prove ¥\X\ = o(l) we will derive a strong upper bound for the probability in ^ when r ^ f^. Specifically, 
we will prove that Pr[0 is a-coreless | is satisfying] < e"-'^^'')" for a function / such that for all r > c^. 

By (|5]l, for all such r we have K[X] = o(l) and Theorem[T3]follows. 



5.1 Coarsening as Hypergraph Stripping 

Given any CNF formula F and any a G S{F) it is easy to see that w(cr) is completely determined by the set of clauses 
U{a) that have precisely one satisfied literal under a. This is because after any sequence of coarsening steps applied 
to (J, a clause that had two or more satisfied literals under a, will have at least one satisfied literal or at least two * and 
thus never prevent any variable from being free. Therefore, to coarsen a truth assignment a it is enough to consider 
the clauses in U{(7). Let us say that a variable v is unfrozen if there is no clause in which it is the unique satisfying 
variable and let us say that a clause is unfrozen if it contains an unfrozen variable. It is now easy to see that coarsening 
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cr is equivalent to starting with U and removing unfrozen clauses, one by one, in an arbitrary order until a fixed point 
is reached, i.e., no unfrozen clauses remain. Variables occurring in any remaining (frozen) clauses are, thus, frozen in 
w(o') (to their value in a), while all other variables are assigned *. This view of coarsening as repeated removal of 
clauses from U {a) will be very useful in our probabilistic analysis below. 

To estimate Pr[0 is a-coreless | is satisfying] we consider a random fc-CNF formula with rn clauses chosen 
uniformly among those satisfying 0. To determine w{0), by our discussion above, it suffices to consider the clauses in 
our formula that have precisely one satisfied (negative) literal. The number of such clauses is distributed as 



It will be convenient to work in a model where each of these m clauses is formed by choosing 1 negative literal 
and k—1 positive literals, uniformly, independently and with replacement. (Since m = 0{n), by standard arguments, 
our results then apply when replacement is not allowed and the original number of clauses is rn — o{n).) We think of 
the k literals in each clause as k balls; we paint the single satisfied literal of each clause red, and the fc — 1 unsatisfied 
literals blue. We also have one bin for each of the n variables and we place each literal in the bin of its underlying 
variable. We will use the term "blue bin" to refer to a bin that has at least one blue ball and no red balls. With this 
picture in mind, we see that the *-variables in w{0) correspond precisely to the set of empty bins when the following 
process terminates: 

1. Let V be any blue bin; if none exists exit. %Identify an unfrozen variable v if one exists. 

2. Remove any ball from v. %Remove the occurrence ofv in some (unfrozen) clause c. 

3. Remove fc — 2 random blue balls. %Remove the other k — 2 unsatisfied literals of c. 

4. Remove a random red ball. %Remove the satisfied literal in c. 

Note that the above process removes exactly one clause (1 red ball and fc — 1 blue balls) in each step and, therefore, if 
we pass the condition in Step 1, there are always suitable balls to remove. To give a lower bound on the probability that 
the process exits before m steps (thus, reaching a non-trivial fixed point), we will give a lower bound on the probability 
that it exits within the first i = am steps, for some carefully chosen a — a{k, r) G (0, 1). In particular, observe that 
for the process to not exit within the first i steps it must be that: 



To bound the probability of the event in (H)) we will bound the probability it occurs in the following simplified 
process. The point is that this modified process is significantly easier to analyze, while the event in (O is only 
(slightly) more likely (for the values of k, r of interest to us). 

(a) Let V be any blue bin; if none exists go to Step (c). 

(b) Remove any ball from v. 

(c) Remove a random red ball. 

Lemma 15. The event in ^ is no less likely in the modified process than in the original process. 

We prove Lemma [15] below. To bound the probability of the event in ^ in the modified process we argue as 
follows. Let q be the number of bins which do not contain any red ball after i steps and let b be the original number of 
blue balls in these q bins. If b < i, then after b steps of the modified process every non-empty bin will contain at least 
one red ball, since up to that point we remove precisely one blue ball per step. Therefore, the probability of the event 
in (HI) is bounded above by the probability that b > i. To bound this last probability we observe that the red balls in 
the modified process evolve completely independently of the blue balls. Moreover, since we remove exactly one red 
ball in each step, the state of the red balls after i steps is distributed exactly as if we had simply thrown m — i red balls 
into the n bins. 




At the beginning of each of the first i steps there is at least one blue bin. 



(8) 
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So, all in all, given a random fc-CNF formula F with rn clauses and a fixed < i < n, conditional on satisfying 
F, the probability that the coarsening process started at fails to reach a fixed point within i steps is bounded by the 
probability that h>i, where 

b = Bin {{k - l)m, , where (9) 
/ k \ 

m = Bm rn, —r , and (10) 



2'^- - 1 ^ 

q = Occ(m — i,7i) , (11) 

where Occ(x, y) is the distribution of the number of empty bins when we throw x balls into y bins. 

As a result, given k,r, our goal is to determine a value for i that minimizes Pr[b > i]. Before we delve into 
the probabilistic calculations, in the next section we comment on how our analysis relates to the planted assignment 
problem and to the existence of non-trivial cores for small values of k. 

Proof of Lemma\T5\ Consider a process which is identical to the original process except with Step[3]removed. We will 
call this the intermediate process. We begin by proving that the original and the intermediate processes can be coupled 
so that whenever the event in (O occurs in the original process it also occurs in the intermediate process. 

First, observe that the evolution of the red balls in both processes is purely random and therefore can be assumed to 
be identical, i.e., we can think of the original process as making a genuine random choice in Step|4]and the intermediate 
process as mimicking that choice. (We think of all balls as carrying a distinct identifier.) Similarly, we can assume 
that originally, the placement of the blue balls in bins is identical for the two processes. 

Let us say that a pair of blue ball placements is good if in every bin the set of blue balls in the original process is a 
subset of the set of blue balls in the intermediate process. Clearly, whenever we are in a good configuration, since the 
placement of the red balls is identical in the two processes, any choice of bin and ball of the original process in Steps 
1,2, is an available choice for the intermediate process. Moreover, if the intermediate process mimics these choices, 
this results is a new good pair of blue ball placements. By induction, since the original pair of blue ball placements is 
good, if the event in dHJ occurs in the original process it also occurs in the intermediate process. 

Next, we compare the intermediate process to the modified process observing that they are identical except that in 
the event that we run out of bins containing only blue balls the intermediate process stops, while the modified process 
carries on. Therefore, we couple the two as follows: the modified process mimics the intermediate process for as long 
as the event in ^ does not occur, and makes its own random choices afterwards. Therefore, if the event in ^ occurs 
in the intermediate process it also occurs in the modified process. □ 



6 The planted assignment model and small values of k 

Conditional on being satisfying, analyzing w{0) is exactly the same as working in the "planted assignment model" 
and analyzing the core of the cluster containing the planted assignment. This is rather easy to do if we are content with 
results holding with probability 1 — o(l). Specifically, by (l9l),(fT0]i,(fTni and standard concentration results it follows 
immediately that if i = am then w.h.p. 

m = \ ■ n + o{n), where A = —j^ (12) 

g = 7 • 71 + o(n), where 7 = cxp (— A(l — a)) (13) 
b = (3-n + o{n), where /3 = (A: - 1)7A . (14) 

With these conditionals in place, we can next determine the mean path of the coarsening process using the method 
of differential equations 1241 . i.e., the number of red and blue balls after each step, up to o{n). In particular, this allows 
us to show that 

Claim 16. For every k > 3, there exists a critical value t\ such that if r < tj. then w.h.p. w(0) — (*,...,*), while if 
r > tl, then w.h.p. a bounded fraction of the variables in w{Q), and therefore in C(0), are frozen. 
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k 



3 



4 



5 



6 



7 



Lower 
Upper 



3.52 
4.51 
5.72 
6.25 



7.91 
10.23 
11.58 
12.34 



18.79 
21.33 
21.75 
22.90 



40.62 
43.51 
40.13 
41.95 



84.82 
87.88 
73.88 
76.84 



In the table below we give the value of for some small values of k (rounding to two decimals). By lower/upper 
below we mean the best known lower/upper bound for satisfiability threshold. 

We see that for k = 3, 4, 5, the probability that has a non-trivial coarsening fixed point conditional on being 
satisfying, tends to for all densities in the satisfiable regime. Clearly, conditioning on "0 is satisfying", is not the 
same as picking a "typical" satisfying assignment. Nevertheless, the gap between <J, and the best threshold upper bound 
for fc = 3 is sufficiently large to strongly suggest that below the satisfiability threshold, most satisfying assignments 
do arrive at (*,... , *) upon coarsening. This is consistent with the experimental results of |fT9ll , who first raised this 
possibility. That said, a distinction worth mentioning is that even if the coarsening procedure arrives at (*,..., *) from 
most/all satisfying assignments there can still be (many) frozen variables: simply, their corresponding clusters may 
not be be compact ("cube-like") enough for coarsening to discover their core. 

We now comment on the couple of simplifications of the original process that we introduced in the previous section 
in order to get a process that is easier to analyze. As we showed, these simplifications only increase the probability of 
the event in ^ and it is natural to wonder if this increase is significant, allowing for the possibility that our analysis 
can be made much sharper Below we give evidence that this is not the case. In particular, if each of m, q, b can be 
assumed to be within o{ti) of its expected value, then the inequality b> iin the modified process is equivalent to 



In the table above we give the value of u/, = min^ Uk{a) for some small values of k. As we can see, these values 
are quite close to and get relatively closer as k is increased. In other words, considering the modified process does 
not cause too big a loss in the analysis. Indeed, taking e.g., a = 1/ In A:, already gives Uk (2''/fc) In fc, which is 
consistent with the physics prediction that tj. (2'^/fc) In fc. 

Of course, if one is interested in establishing that certain properties of w{0) hold with exponentially small failure 
probability, as we do, then conditioning that m, q, b are within o{n) of their expectation is not an option. One has to 
do a large deviation analysis of all these variables and their interactions in the coarsening process and determine the 
dominant source of fluctuations. This is precisely what we do with respect to the event b > iin the modified process. 

7 Large Deviations 

It is well-known that if Tip > then for every S > —1, 



F{x, y) = cxp(-j;[(l + y) ln(l + y) - y]) . 

A similar large deviations bound was shown in [151 for the number of empty bins in a balls-in-bins experiment 
(Theorem 3). That is, for every 6 > —1, 




Pr[Bin(n,p) 



{l + 5)np] < F{np,S) , 



where 



Pr[Occ(TO,n) = (1 + S)e 



— m/n' 



] < F{ne 



~7n/n 
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7.1 Application 

Write r = A(2'^ — l)/fc and fix S,e,C, > —1. Write p = A(l + S){1 — a) in order to compress the expressions below. 
The probability that 

m = (1 + 5)E[Tn] = {1 + S) J"^ ■ n = {I + 6)X ■ n , (15) 

2" — 1 

/ Til — ? \ 

q = (l + C)IE[g|m] = (1 + C)expf _ j . n = (1 + Oe"" • n , (16) 

h = {l + e)E[b\q,m]^{l + e){k-l)m-^ = {l + 6){l + e){l + C)X{k-l)e-''-n, (17) 

n 

is bounded by 

F{Xn,S)-F{e-Pn,(:)-F{{l + S){l + OX{k-l)e-Pn,e) . (18) 
We write this as e~"^, where 

O = XLu{d) + e-Puj{C) + X{k - 1)(1 + S){1 + C)e"''w(e) , 

with uj{x) = (1 + x) ln(l + x) — x. 

Conditional on the events in (fT5]l-(fT7]i we see from ( [TT] ) that the condition b> i becomes B >Q, where 

B = (l + e)(l + C)(fc-l)e-''-a . 

For any fixed k, r and a define $ = {{S, C,^) ■ B > 0}. Thus, 

Pr[0 is a-coreless I is satisfying] < exp(— n • minil) x poly(n) 

and to prove that the expected number of a-coreless assignments in o(l), it suffices to prove 

minfi > ln2 + rln(l - 2"'') = s . (19) 

8 Optimization 

To establish ( fT9l ) it is enough to prove that the maximum of B in the variables 6, ^ and e under the condition il < s 
is negative. Considering that the function B is monotone in the three variables 5, ( and e, the maximizer of B in the 
region il < s has to be on the boundary, that is for fl = s. The maximum of B under the condition il = s corresponds 
to the extremum of the function G — B — p,{^l — s), where /i is a Lagrange multiplier The equations for the location 
of the maximizer are thus given by derivatives of G with respect to S, C e and jj, 



dsG = ^ dsB = i^idsn (20) 

9cG = => dQB = pdQn (21) 

= ^ d^B = ^id^n (22) 

d^G = => Q = s . (23) 



Lemma 17. For any fixed k, r and a G (0, 1), at the extremum of the function G defined by equations ( I20| )-( I23I ) the 
following assertions hold 

1. e is non-negative; 

2. Q is non-negative; 

3. 5 is non-positive; 
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Proof. The first assertion follows by observing that B is an increasing function of e and Vl contains e only in the third 
term through Lo{e). Therefore, if the maximizer would be in e = e' < 0, moving to e = e" > 0, with uj{e') = t^(e")> 
would keep constant while increasing B. 

Combining equations ( 1211 1 and ( l22b in order to remove ji we have 

dc^n = d^n di^B/d^B 

that is 

e-P ln(l + + (1 + 5)X{k - \)e-Puj{€) = (1 + 5)X{k - l)e-P{l + e) ln(l + e) 
which, after simplification, reduces to 

ln(l + C) = {k-\)\{\ + 6)e . (24) 

Thus, for (5 > — 1 and e > we have that at the maximizer C > 0, proving the second assertion. 
Combining equations ( l20l i and ( l22b we can write 

Q = dsn- d,n dsB/d,B = Ae"" p ln(l + <5) - (1 - a)uj{C) - p(l + C}(k - l)t^(e) + 

+ {k~ 1)(1 + CV(e) + p(l + e)(l + C)(fc - 1) ln(l + e) 
The term within square brackets can be simplified to 

e" ln(l + S)-{1- a)uj{C) + p{l + C){k - l)e + (fc - 1)(1 + C)^(e) 
which, using ( l24b . implies 

e" ln(l + (5) + (1 - a)C + (fc - 1)(1 + CV(e) = . 

Since for C, e ^ the second and third term terms of this expression are non-negative, we find that 5 has to be 
non-positive at the maximizer in order to satisfy the last equation (third assertion). □ 

We next prove some bounds on S and e, that hold at the maximizer. 

Lemma 18. Fix any r, k, and a e (0, 1). At the maximizer of B, 

^<S<0. 
A 

Proof. Since S in non-positive at the maximizer, we observe that ll!{6) > 5^/2 for 6 < 0. Moreover each of the three 
terms in ft is non-negative for (5, C > —1 and this implies 

Xoj{6) < s =^ X— < s =^ \S\ < . 

2 V A 



□ 



Lemma 19. Fix any r, k, and a G (0, 1). At the maximizer of B, 



1 — a In 3 _ 

' ^ "fc^ ^ A(l + (5o)(A:-l) " '° " 

Proof. Since every term in Vi is non-negative, considering the second term, and using the fact s = we get 

s > ujiOe-P . 

Observe now that for all x > —1, we have uj{x) > x ~ 1. Therefore, 

se" + 2 > 1 + C . (25) 
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Using ( l24b to replace 1 + C and the fact s < In 2 < 1 we can conclude from dZST l that 

^A(i+d-)(fc-i)e ^ g ^ In (eP + 2) ^ p + ln3 



2 > e-" 



A(l + (5)(fc-l) - A(l + (5)(fc-l) ' 
where for the last inequality we use that p = A(l + — a) is non-negative. We conclude that at the maximizer 

1 — a In 3 



e < 



A: - 1 A(l + (5o)(A:- 1) 
where 5 has been replaced by its lower bound value. 



□ 



Thus, the stationary point of G must occur in the region A = {((5, e) : (5o < <5 < 0, < e < eg}. In the next 
subsections we derive analytical results for this optimization for all k > 14, and we summarize results obtained by 
numerically finding the stationary point of G for 9 < fc < 13. 

8.1 Proving the existence of frozen variables for A; > 14 analytically 

For any fixed values of d and e, the requirement B > implies 



c> 



1 



(k-m + e) 

Plugging this lower bound in the second term of il, we see that the requirement il = s impHes 



In 



(k-m + e) 



- 1 



-p < 



(26) 



(27) 



Therefore, it suffices to find A and a such that dZTl l cannot be satisfied by any Sq < S < and < e < eo. This is 
certainly true if a lower bound to the l.h.s. of (|27] | makes such an equation unsatisfied, that is if 



In 



a e 



A(1+5o)(1-q) 



1 



(fc-l)(l + eo)L V(fc-l)(l + eo), 
and the term within the squared brackets above is positive. Thus, to summarize, it suffices to find A and a such that 

,A(1+5o)(1-q)- 



(fc-l)(l + eo) 



In 



a e 



1 



> s 



p-A(l-a) ^ Q 



,(fc-l)(l + eo). 
With the change of variable A = c fc In 2 we have that 

s = ln2 + c(2'= - l)ln(l - 2"'') In 2 < (l - c(l - 2"'')) In 2 , 

and 

s - e-^(i"") < (1 - c(l - 2-'^')) In 2 - 2^^'=(i-") < (1 - c) In 2 , 
since for any c G [0, 1], a E [0, 1] and fc > 

cln2 < 2'=(i-'=(i-")) . 



(28) 



Therefore, it suffices to establish 



a 



(fc - l)(l + eo) 



In 



a e 



cfc(l+5o)(l-C()ln2 



(fc-l)(l + eo) 



> (1 - c)ln2 . 



(29) 
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Based on Lemmata [TS] ancfT9l we now introduce simpler bounds for S and e, which hold for all c > 4/5 and k > 2. 
Specifically, 



and 



1 



fc- 1 

Replacing dSOb and dSTT ) in ( |29l ) we have 



1 - a + 



In 3 



c/c(l + (5o)ln2 



< 



a 



k + 1 



ck(l-a){l- ^1 ln2 + ln ( 



- 1 



k-1 



- (1 - c)ln2 > . 



Solving with respect to c, the last inequality becomes 



c > 



k+l 



1 - In 



fe+i 



/In 2 



1 + -")TTT7r 



= gc{k,a) 



For any fixed a e (0, 1), gc{k, a) is a decreasing function of fc, which as fc ^ cx) tends to 

1 



1 + a(l - a) 



(30) 



(31) 



In order to prove that there exists a choice of a such that min$ VL > s for some r < rk and all fc > fco(a), we 
rescale the lower bound for ru from ([TJ as 



2'= In 2 



Tk 



{k+l) In 2+3 
2 



(2'= - l)ln2 



1 - 2 



-fc 



(fc + l)ln2 + 3 
2(2*^ - l)ln2 



and observe that is increasing in fc. In Figure|2]we now see that the function 5c(14, a) dips below T14 for a certain 
range of a, implying that the left endpoint of this range is an upper bound on the fraction of unfrozen clauses. For 
larger values of fc things only get better since gc{k, a) is monotonically decreasing with fc, whereas is increasing. 
For any fixed value of a, fco(a) can be defined as the first fc value for which gdk, a) < holds. 



1.002 



.998 



. 996 



. 994 




£U 4 



Figure 2: The function gc{k,a) for fc = 14 as a function of a. The horizontal line, slightly below 1, is T14 
0.9994711565304686. 
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8.2 The A; = 9 case 



Recall that for any fixed fc, r and a the function G depends on four variables: 5, e, C and /i. We will plot G for fc = 9, 
a = 0.265 and a few different values of r, while fixing C, and /^j at the value they take at the stationary point: Q is given 
readily by (l24l i. and substituting this value of C into (|22] | we get 

1 

A(l + <5)ln(l + e) " 

In the upper left panel of Figure|3]we show G in the subregion of A corresponding to the optimal C,, ji for r = 347. 
By closer inspection one finds that there is a unique stationary point in this region. The remaining three plots are 
zoomed on the stationary point for r ~ 347, r = 347.5 and r = 348. It is clear that the function G at the stationary 
point is positive for the first two r values and negative for the third one (for the sake of clearness, negative values of 
G are not plotted). Thus, for k ~ % and a — 0.265, the critical value of r lies between 347.5 and 348. In the next 
subsection we determine this critical value numerically for all 9 < fc < 13. 




Figure 3: fc = 9, a = 0.265 

8.3 Optimizing for 9 < < 13 

For any fixed fc and a the value of c^, such that w.h.p. clustering exists for r > cJJ, can be computed by solving 
numerically (|20]i-(|23Tl together with G = [which reduces to B = since 51 = s by (|23]l1. Adding a sixth equation 
daG = allows one to minimize with respect to a (at some a„i) thus determining the smallest density Ck for which 
the existence of frozen variables can be established. Numerical solutions of these six equations are given in the table 
below for 9 < fc < 13 along with the lower bound from ([T]i- 



fc 




Cfc 


am 




S 


c 


e 


9 


349.92 


347.84 


0.265 


8.037 


-0.015085 


1.7336 


0.02083 


10 


704.94 


690.48 


0.273 


6.935 


-0.015714 


2.7134 


0.02194 


11 


1413.90 


1370.42 


0.281 


6.256 


-0.015789 


4.0330 


0.02229 


12 


2833.12 


2720.44 


0.289 


5.802 


-0.015548 


5.7977 


0.02220 


13 


5671.90 


5402.23 


0.297 


5.480 


-0.015132 


8.1457 


0.02184 
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9 The existence of cluster regions 



In this section we prove the existence of ak, Pk as in Theorem|9] Let 

h{x) = —X ln.T — (1 — x) ln(l — x) 

< ln2-2(l/2-x)2 , for any xe [0,1]. 

We begin by bounding In A from above as follows, 

lnA(a,fc,72'^ln2) = ln2 + /i(a) + 72*^ ln2 In [l - 2^-'^ + 2-'=(l - a)''] 
< 21n2 -2 (1/2 - a)^ -7ln2[2- (!-«)*=] 
= ■w{a,k,j) . 

We note that for any fixed fc, 7, the function w{a, k, 7) is non-increasing in k and decreasing in 7. Moreover, 



^ --f\ii2k{k-l){k-2){l-ar~'^ <0 



(32) 



implying that for any fixed k, 7, the equation w{a, k, 7) = can have at most three roots for a e (0, 1). To bound the 
location of these roots we observe that for any fc > 8 and 7 > 2/3, 



u;(0,fc,7) - (2-7)ln2- - >0 , 

u;(l/2,A:,7) = [2 - (2 - 2-'^)7] In 2 > , 
w(99/100, k, 7) < w(99/100, 8,2/3) = -0.0181019... < , 



(33) 

(34) 
(35) 



where the inequality in ( [35] l relies on the mononicity of w in A:, 7. Therefore, from (|33]l-(l35]l we can conclude that 
for every A: > 8 and 7 > 2/3, if there exist a^, /3fe G (0, 1/2) such that w{ak, k,j)<0 and w{f3k, k, 7) < 0, then 
A{a, k, 72*^ In 2) < 1 for all a S [ak,l3k]- Below we first prove that such ak,(3k exist for all A: > 8 and then prove 
that for sufficiently large k, we can take ak,Pk as in (O. 

• For fc = 8 it is enough to consider the plot of A(q;, 8, 169) in Figure[Tl For fc > 9 we take 7 = 0.985 > 2/3. 
Note that 0.985 • 2*^ In 2 is smaller than the lower bound for given in ([T]), for all A: > 9. 

- We take ak = l/k. We note that w(l/9, 9, 0.985) = -0.0451... < and prove that w(l/A:,fc,7) is 
decreasing in k for any A; > 4 and 7 < 1 as follows. 



dw{l/k, k, 7) 
dk 



= 7 In 2(^1 
< 7 In 2^1 
In 2 



fe-i 



fc-i 



1 

_k 
1 

fc2 



1 



4 

fc2 



In 1 



4 



< 



1 

< . 



- We take /3k = 3/8. We note that w{a, A;, 7) is non-increasing in k when a and 7 are fixed and that 
w(3/8,9,0.985) = -0.000520265... < 0. 

• For the setting where r = (1 — 6)2'' In 2, we will additionally use that —2a; In 2 < ln(l — x) < —x for all 
< X < 1/2 to establish that for any 1 < c < A;/2, 



lnA(c/A:,A;,r) = In 2 + /i(c/A;) + r ln(l - 2^-''' + 2-'=(l - c/A;)'^') 

< ln2+(c/A:)(lnfc + 21n2)-r(2i-'^'-2-'^(l-c/A-)'') . 



(36) 
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Substituting r = 72'^ In 2 into ( |36] ) we get 

lnA(c/A:,fc,72'=ln2) <ln2(l-27 + 7e"=) + (c/fc)(ln/c + 21n2) . (37) 

- If c = 1 and 7 > 2-\/e ~ 0-612..., then (|37] | implies that In A(l/fc, fc, 7) < for all sufficiently large k. 

- If 7 = (1 -5) > 2/3, then for any 1 < A < 3/(4 In 2) = 1.082... 

w{l/2-VX6 \n2,k,l-S) = -2(A - 1)(5 In 2 + (1 - (5) In 2 Q + VA(51n2^ , 

which is negative for all sufficiently large fc. The choice f3k = 1/2 ~ (5/6) a/J corresponds to A = 
(5/6)^/ In 2 = 1.00187..., which is a valid value. For k large enough we have ak = 1/fc < Pk — 
1 /2 - 5 V^/6 for any S e {0,1 /3). 



10 The existence of exponentially many cluster regions 

We will use the following two lemmata. 

Lemma 20. Ifj > 49/50 and k > 11, or-fE (2/3, 1) and k > 15, 

9 In 2' 



In g{k, 72*= In 2) < (1 - 7) In 2 + 1 



16 



(38) 



Lemma 21. For all k > 

where 

m{k) = 1 + 



2fc + 3 ,,fc 



lnAb(l/2,fc,72''ln2) > 21n2[l - 7m(fc)] , 



3fc^ + 6fc-4 ^_,,. ^ 13fc-^-12fc+l ^_3, 



(6fc^ - 13fc2 + 2fc)2 



-4fe 



9k^ - 24fc3 + I0fc2 



,-5fe 



+ (9fc"* - 6fc3)2-6fe + Zfc42-7fc 



Combining the two lemmata above we get that if r = 72*^ In 2 and either 7 ~ 49/50 and fc > 11, or 7 e (2/3, 1) 
and fc > 15, then 



log2 



Afc(l/2,fc,r) 
g{k,r) 



1/2- 



> 



1 



2 In 2 



ln2(l + 7- 27m(fc)) - 1 + 



9 In 2 



16 



(39) 



where m{k) is as in Lemma 1211 It is not hard to check that m(fc) is decreasing in fc. 

• For 8 < fc < 12, the existence of > can be verified by plotting A and A;, and noting that 

Af,(l/2, fc, r) > max A(a, fc,r) , 
Qe[o,A] 

both when fc = 8 and r = 169 and when 9 < fc < 12 and r = 0.985 • 2'' In 2. For fc > 12 and 7 = 0.985, the 
existence of > follows from the fact that the expression inside the square brackets in ( [39] l is positive when 
fc = 13 and 7 ~ 0.985 and m(fc) is decreasing in fc. 

• For the setting where r = (1 — 6)2'^ In 2, we note that the limit of the expression inside the square brackets 
in ( [39] l as fc ^ 00 is (1 — 7)/2. In particular, writing r = (1 — 6)2'' In 2, it is not hard to show that the right 
hand side of (|39]l is greater than 6/2 - 3/fc^ for all fc > ko{6). 
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10.1 Proof of Lemma IIS The volume of the largest cluster 

Below, we consider k and r to be fixed, so tliat all derivatives are with respect to a. Specifically, we will give i) a value 
aM such that A is non-increasing in (oa/, ctk) and ii) a function u which is non-decreasing in [0, aAi) and for which 
A(a, k, r) < u{a, k, r). Thus, we will conclude g(fc, r) < u{aM, k, r). 
We begin by getting an upper bound for A', as follows: 

.// , N , , , N k(l~a)^~^ 
A (a,k,r) = - Ina + ln(l - a) - — — —, 

< -Ina - a -2^''rk{l - a)""^^ 

< - In a - 2"''rfc(l - a)''-^ (40) 

< -\na-2-''rk{l~ ka) 
= u{a,k,r) . 

Lemma 22. Ifr = 72'' In 2, then for all k > 8 and 7 > 3fc^^ log2 k, there exists 

aM < 2-^'=(l + 47/fc22-T'=ln2) , (41) 

such that u{aMi r) — 0. 
Proof of Lemma\22\ Let 

q{a) = 2-^^2^^''°' . 

We begin by noting that if a^i is such that q{aM) = ct^ then u{aM ,k,r) =0. Now, let us define 

s{a) = 2-''''{l + 2a-fk^h-i2) . 
Observe that the unique solution of s{a) — a is 

2-7fe 

a* = (42) 

I -2"fk^2~^''\n2 

and that s{a) > a for all a £ [0, a*). 

Recall that < l + 2x for all < x < 1. Therefore, q{a) < s{a) for all a such that 'yk'^a In 2 < 1. In particular, 
if 'yk'^a* ln2 < 1, then since s{a) > a for all a £ [0, a*), we can conclude that the equation q{a) = a has at least 
one root aM < a*, as desired. 

By (|42] |. the condition •yk'^a* In 2 < 1 is equivalent to 

lk^2-'''' < = 0.4808... (43) 

To establish that (03]) holds we note that for any 7 > 3fc^^ log2 k the quantity ^k'^2^^^ is decreasing in 7 and, 
therefore, it is bounded by z{k) = 3fc~^ log k. As z{k) is decreasing for k > 2, for all fc > 8 we have ^k'^2~'^^ < 
z{8) = 9/64 = 0.1406... < 0.4808..., as desired. The fact 7fc22-'>''=^^ < 0.1406... along with the inequality < 
1 + 2a: valid for a; < 1/2, gives us q;a/ < a* < 2-'''''(l + 47^22^''''' In 2). □ 

To bound A by an non-decreasing function we note 

In A(a, fc,r) < ln2 - alna + a - r2"''(l + a) = M(a, /c, r) . (44) 
Lemma 23. Ifr = ^2^' In 2, then for every fc > 8 and 7 G (3fc~^ logj fc, 1], 

"(aM,fc,r) < (l-7)ln2+ M fc-2 . 
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Proof. Using Lemma l22l to pass from (l45T l to ( |46] |, we see that for every fc > 8 and 7 € (3fc ^ logj fc, 1], 

ii(Q;M,fc,?') = ln2 + Q!A/(7fcln2 — 7fc^Q;jv/ ln2) + q;a/ — 7ln2(l + q;m) 

< (l-7)ln2 + aA/[l + 7(^--l)ln2] 

< (l-7)ln2 + 2-'''^(l + 47fc22-'''^-ln2)(7fcln2 + l) . 



(45) 
(46) 



Recalling that (|43] | holds for all fc > 8 and 7 > 3fc ^ log2 fc, we conclude 



u{aM,k,r) < (l-7)ln2 + fc-3 
< (l-7)ln2+(l- 



9 In 2 



16 



(fcln2 + l) 



9 In 2 
16 



□ 



Ina — r2 it 



We can now prove Lemmal20l 

Proof of Lemma^U\ Recall the definition of the function u from ( |44] | and note that, since u'{a) 
is non-decreasing for r < 2*^ and a < 1/e. From (HTI) we see that < 1/e and therefore we can conclude that 
A{a, fc, r) < u{aM, fc, J') for all a G [0, aA/)- To complete the proof it thus suffices to prove that A is non-increasing 
in the interval (aA/, 1/fc) since, by our results in the previous section, we know that A < 1/fc both when 7 > 49/50 
and fc > 11, and when 7 <E (2/3, 1) and fc > 15. For that we first observe that 



u' (a, fc, r) 



1 

a 



1 

a 



Since, by definition, u(q!a/, k,r) = this implies ti < for all a e [aA/, 1/fc^] and since A' < ii, it follows that 
A' < also for such a. Using ( l40l i. it is straightforward to check that for a G [1/fc^, 1/fc], the derivative of A is 
negative both when i) 7 > 49/50 and fc > 11, and when ii) 2/3 < 7 < 1 and fc > 15, thus concluding the proof. □ 

10.2 Proof of Lemma 111]: A lower bound on the number of balanced assignments 

Proof of Lemma^J] Recalling the definition of Ab from [5| we have 



lnAb(l/2,fc,r) = 21n2 + rln 



((l-e/2)* 



(1-e)* 



where e satisfies 



e(2-e)'=-i = 1 
We note for later use that, as shown in ||5l, if e satisfies ( |48] ) then 

2i-'= + fc4-'^ < e < 2^-''- 



3fc4 



-fc 



Since all coefficients in the binomial expansion of (1 — e) are positive 



(1-e)-'^ > 1 + fce 



fc(fc+l) ,2 



(47) 



(48) 



(49) 



(50) 



To get a lower bound for the numerator inside the logarithm in ( |47] | we consider the binomial expansion of (1 — e/2)'^. 
We observe that the sum of a pair of successive terms where the lower term corresponds to an even power equals 



fc 



ie/2y 



1 



ik-j)e 



2(J + 1) 



(51) 
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For > 8, j > 4 and e <h/2 the expression in ( fSTT l is positive. Moreover, when k is even the last term in the binomial 
expansion has a positive coefficient and can be safely discarded. Therefore, for all fc > 8 and e < 5/2, 

, fee fc(fc-l)e2 l)(fc-2)e3 
(1 ~ e/2f > 1 - y + ^ ^ — ■ (52) 

Substituting dSOl l and (|52] | into (|47] | we get a lower bound of the form In Ab > co + cie + C2e^ • • • + cse^. It is 
not hard to check directly that cg > for all k > 8. Similarly, using the upper bound for e from i49[ . it is not hard to 
check that for i = 2, 4, 6, we have a + Ci+ie > for all fc > 8. Therefore, we can conclude 

lnAfc(l/2,fc,r) > 2 In 2 + r In [l - 2^-'' + 2-2'^' - efc2-'''(l - 2"'')] 

> 21n2 + rln[l-2i"'' + 2"2'=-fc2"'^'(l-2"'')(2^"'' + 3fc2~2'=)] , (53) 

where in ( |53] ) we have replaced e with its upper bound from ( |49] l. 

The argument of the logarithm in (l53T l is increasing in k for all A: > 3 (a fact that can be easily established by 
considering its derivative). As a result, we have that for all fc > 8, it is at least equal to its value for fc = 8 which is 
1 — 0.00805183... > 1/2. Thus, using the inequality ln(l + x) > x — valid for all x > —1/2, we can finally write 



where 



lnAb(l/2,fc,72'^'ln2) > 2 In 2[1 - 7m(fc)] , 



2fc + 3„_fc 3fc2 + 6fc-4 2fc 13fc^ - 12fc+ l„_3fc 



m(fc) = 1 + ^—2-" + 2-^^- + 2 

+ (6fc3 - 13fc2 + 2fc)2-4fe + 9fc"-24fc3 + 10fc^ ^_5fc ^ ^^^4 _ g^3)2-6fc ^ 9^.42-7^ (54) 



2 



□ 
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